Orbital order in Mott insulators of spinless p-band fermions 
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A gas of strongly interacting spinless p-orbital fermionic atoms in 2D optical lattices is proposed 
and studied. Several interesting new features are found. In the Mott limit on a square lattice, 
the gas is found to be described effectively by an orbital exchange Hamiltonian equivalent to a 
pseudospin-1/2 XXZ model. For a triangular, honeycomb, or Kagome lattice, the orbital exchange 
is geometrically frustrated and described by a new quantum 120° model. We determine the orbital 
ordering on the Kagome lattice, and show how orbital wave fluctuations select ground states via 
the order by disorder mechanism for the honeycomb lattice. We discuss experimental signatures of 
various orbital ordering. 



The electron orbital degree of freedom plays an impor- 
tant role in correlated quantum materials such as tran- 
sition metal oxides [1]. Many intriguing phases observed 
in experiments are attributed to the coupling of elec- 
tron d-orbitals, typically in the t2g or Cg manifold, to 
the electron charge, spin, and/or the lattice degree of 
freedom 0, Q ■ Understanding the intricate interplay be- 
tween them remains a theoretical challenge. It is thus de- 
sirable to study simpler systems where the orbital degree 
of freedom is disentangled from others, namely "plain 
vanilla" orbital ordering in a Mott insulator. 

Rapid advances in loading and controllingalkali atoms 
on the excited bands of optical lattices 0, 0] have 

' made it possible to investigate orbital ordering of bosons 
and fermions in new settings 0, [l^, [ll|, [13, [13, 

\ Here we show that strongly interacting single species 
(spinless) p-orbital fermions on optical lattices can real- 
ize Mott states characterized by orbital-only models [ll] . 
We consider the case where the degenerate p-orbitals of 
the optical potential well are partially occupied while the 
fully occupied s-orbital acts as "closed shell" and remain 

\ inert at low energy scales. Thus, these atomic Mott in- 
sulators differ significantly from their solid state coun- 

' terparts of d electron systems in orbital symmetry. We 
derive the low energy effective orbital exchange Hamil- 

\ tonian of p-band fermions in the strong coupling regime 
and commensurate filling for simple 2D lattices and de- 

' termine their long range orbital ordering patterns. 

Mott states of p-band fermions, described by orbital- 
only models, represent new examples of correlated states 
in condensed matter. Although we refer to the emergent 
low energy symmetry of the degenerate p-orbital states as 
pseudo-spin symmetry, it is not an internal symmetry like 
the true spin degree of freedom, but rather intrinsically 
spatial. As we shall demonstrate, in general Px,y orbital 
states transform as the components of a spinor under 
space rotation, in a manner reminiscent of the Lorentz 
spinor. This sets the orbital exchange models apart from 
the familiar spin exchange models. 

Spinless lattice p-band fermions. We consider a gas of 
fermion atoms in a single hyperfine spin state loaded in 
an optical lattice interacting via a generic two-body po- 



tential yiri — r2). Atoms interact predominately in the 
p-wave channel, i.e., in momentum space, l^(k' — k) ~ 
3ti{k){k' ■ k). The actual form of fiik) is unimpor- 
tant for our discussion [l6| . as long as it reproduces 
the low energy p-wave scattering amplitude /i(fc) — 
k'^/{'~v^^ + cik^ — ifc"^), where vi is the scattering vol- 
ume and ci is the effective range parameter [13] • These 
parameters are known for ^''K jlSj. For example, one can 
use a separable model potential [l6| or a pseudopotential 
[fit with coupling constant gp = VlviT^h? /m (m is the 
fermion mass). The strength of "^i can be tuned using 
a p-wave Feshbach resonance 0, IHI, S We only 
consider repulsive interaction here. 

We focus on a setup that captures the physics of the 
orbital symmetry and quantum degeneracy of fermionic 
atoms. We first consider a strongly anisotropic 3D cubic 
optical lattice potential, Vopiy) = J2^=x,y,z sin^(fcLr^) 
with Vx — Vy and Vz ^ where k^ is the wavevec- 
tor of the laser field and we set the lattice spacing to 
be unit, a = n/ki, = 1. In the deep lattice limit, 
^ En = lT?k1/2m (the recoil energy) for all three 
directions, the bottom of the optical potential at each 
lattice site can be approximated by a harmonic oscillator, 
and the lowest few energy levels are s,Px,Py,Pz orbital 
states. The \px) and \py) orbitals are degenerate, while 
|s) (Ipz)) is well below (above) them, separated in energy 
by the harmornic oscillator frequencies huj^ = ^J4\^pEli. 
Due to the strong lattice potential in z-dimension, the 
system is dynamically separated into a stack of approxi- 
mately independent layers with suppressed tunneling in 
between, each being two-dimensional (2D). Then, in or- 
der to explore the orbital-related quantum dynamics in 
such a lattice, one can fill fermions up to the p-band by 
having an average number of fermions per site between 1 
and 2. In the atomic limit, for two particles per site, one 
of them occupies the s-orbital and the other one occupies 
either px or py. We shall refer to this as half filling. 

We expand the fermion field operator in the Wannier 
basis, tp{T) = ^ - ^ (r — Ri)ci^, where is the lattice 
vector at site i in two dimensions and I — s,Px,Py,Pz, ■■■ 
is the band index. Then the interacting system is de- 
scribed by a multi-band Hubbard model. Close to half 
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filling, with the filled s band and the empty Pz bands well 
separated from the degenerate Px and Py bands, the low 
energy effective model for the fermions is the following 
2D p-band Hubbard Hamiltonian, 

Hp= ^ iMi/[4^c.j+^,p + h.c] + U'^ UixTi^y . (1) 

Here, t^^ = t^^S^^ +^±(1 — is the hopping amplitude 
of orbital fi in the v direction [the orbitals will be denoted 
a,s {px,Py) = [x,y) = (t, i) interchangeably to Hghten the 
notation] , and U is the onsite repulsion between atoms in 
Px and Py orbital states. In the harmonic approximation 
of the Wannier function, the transverse hopping t±^ — 
_g-(')/2) y^j2 is negative and small, while the longitudi- 
nal hopping <|| = (1 — rj^/2)t± is positive and much larger 
in magnitude. Here, = {V^j,/ Efi)^/^kL is the inverse 
of the harmonic oscillator length in the /u-dimension, and 
77 = axd is typically a large number. The onsite energy is 
given by [/ = gpalaz{22al + al)/2,2{2'Kf/'^ in the pseu- 
dopotential approach. Eq.(IT]) looks like the ordinary one- 
band Hubbard model for spinless fermions, albeit with a 
twist: the px (pseudo-spin X) atoms prefer hopping in the 
X direction while the Py (J,) atoms prefer hopping in the 
y direction. Such anisotropy has dramatic consequences 
in the strong coupling limit. 

Effective orbital exchange Hamiltonian. At half filling 
and in the strong coupHng Hmit, C/ » iy, orbital fluctua- 
tion is the only remaining low energy degree of freedom. 
It is well known that virtual hopping processes lead to 
direct and higher order orbital exchange interactions 0. 
In our case, as shown in Fig. [1] the nearest neighbor 
orbital exchange is strongly anisotropic as a direct con- 
sequence of the anisotropic shape of the p-orbitals. The 
effective Hamiltonian for Hp can be derived following the 
standard canonical transformation method. Up to order 
of t|, it is given by i?'^^ = —U^^T^} ^ in the notation of 

Ref. HI- We introduce pseudo-spin operator T+ = cJ,Cy 
and = {c\cx - cJcj^)/2, or equivalently T = ic|^cr^t,c^ 
in the Cartesian vector form with cr being the standard 
Pauli matrices, and rewrite iJ^'^^ as, 

i^orb - ^J^v^T+Tr- + h.c.) + J^T^T^]. (2) 

<ij> 

Replacing T with the usual spin S in a magnetic system, 
Eq.([2]) corresponds to the familiar XX Z model. The 
antiferro-orbital Ising exchange Jz — 2(t5_+i|)/C/ results 
from the longitudinal (and transverse) virtual exchange 
hopping. By contrast, the XY exchange Jxy — 2t±t^^/U 
involves both longitudinal and transverse hopping, it is 
ferro-orbital {Jxy < 0) due to the opposite sign of i|| 
and t±. Because the transverse hopping amplitude of 
orbitals is small, we have \Jxy\ ^ \Jz\, so the leading 
order interaction takes the form of antiferro-orbital Ising 




FIG. 1: (Color online) Left: Virtual hopping processes giving 
rise to the antiferro-orbital Ising exchange Jz and the ferro- 
orbital XY exchange Jxy. The transverse hopping t± is much 
smaller than the longitudinal hopping and has opposite 
sign. Therefore Jxy <C Jz. Dash lines indicate empty orbitals. 
Right: Neel orbital ordering on the square lattice. 

model for pseudospin T = ^, 

Horh — Jz ^ T^T^. (3) 

<hj> 

We neglect three body and ring exchange terms which 
are of higher order in t\\/U. The Ising exchange favors 
antiparallel configuration of nearby pseudospins, i.e. per- 
pendicular configuration of nearby p orbitals (one in px 
and the other py). Thus, the Mott state is antiferro- 
orbitally ordered on square lattice. The translational 
symmetry is broken with an alternative arrangement of 
Px and Py orbitals as shown in Fig. [TJ 

Frustrated 12(F model on oblique lattices. We now 
generalize the analysis to oblique 2D lattices. For any 
vector ee directing at angle 9 with the x axis, such as 
the bond direction indicated in Fig. ^ it is convenient 
to introduce a local coordinate system with axis x and 
y rotated from the global coordinate system by 6. The 
p-orbital wave functions transform under rotation as the 
components of a planar vector, i.e., 

Cx ^ Cx = Cx cos 9 + Cy sin 9 , , , 

Cy ~f Cy — —Cx sin 9 + Cy cos 9. 

Accordingly, the pseudospin operators transform as 

Tz Tx sin 29 + Tz cos 29 , 

Tx -> fx= Tx cos 29 - Tz sin 261 , (5) 

T T — T 

In other words, the pseudospin vector T is rotated by 29 
about the y axis in the pseudospin space. The leading or- 
der exchange interaction between site i and j, connected 
by bond eg, is then given by JzTlTj. Note the value of 
longitudinal hopping along ee , and consequently Jz , for a 
specific lattice depends on details of the optical potential. 

We are particularly interested in the triangular and 
honeycomb lattice, both of which can be implemented 
using interfering laser beams. For these lattices (lattice 
spacing a = 1), we introduce three basis vectors: ei = i. 
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FIG. 2: (Color online) Left: The spatial coordinate system 
(in blue). Basis vectors ej are at 120° from each other. Three 
pseudospins Tj (in black) defined on bond Cj. The quantiza- 
tion axis Tz is a; (p^;). Right: Orbital ordering on an individual 
triangle ABC and long range order on the kagome lattice. 



62, and 63, which are 120° from each other as shown in 
Fig. [5] and use Tj to denote the operator along bond 

e, (j = 1,2,3). Explicitly, Ti = T„ T2 = -\T, - ^T^, 

and T3 = ~2'^z + 2 ^2;, all lying within the xz plane. 
Then the orbital exchange Hamiltonian can be put into 
a simple form. 



H 



120 



J.^T,(R)T,-(R 



(6) 



where the sum over R includes all lattice sites (or the 
sites within the A sublattice) for the triangular (or hon- 
eycomb) lattice. We call Eq. ^ the quantum 120° 
model, as a formally similar Hamiltonian was proposed 
to describe the two-fold degenerate eg electrons in cu- 
bic perovskites [3, [2^, 2^ 27 1. The underlying physics 
is however very different. There, Cj label the cartesian 
basis vectors of the cubic lattice, while the 120° configu- 
ration of three Tj 's arises from a fundamentally different 
transformation property of two cZ-orbitals, |3z^ — r^) and 
\x'^ — y^), under the permutation of x, y, z. 

The antiferro-orbital exchange favors perpendicular 
alignments of nearest neighbor orbitals along bonds. Ap- 
parently, it is impossible to achieve this for all three 
bonds on an elementary triangle. Nor is it possible for 
three bonds joining at a site on the honeycomb lattice. 
The orbital exchange is thus geometrically frustrated on 
the triangular and honeycomb lattice. By frustration, we 
mean the energy on each bond cannot be minimized si- 
multaneously [28|. As we shall show below, the classical 
ground state possesses a continuous degeneracy for the 
honeycomb lattice. Solving the quantum 120° model is 
a nontrivial task, e. g., i t remains an open problem even 
for the cubic lattice 15, 2^, 2^, 27, 2^. Here, in search- 
ing for the ground state ordering pattern, we employ the 
semiclassical analysis which proves fruitful in the study 
of frustrated magnets [s^l : first find the classical ground 
state, then consider the effect of leading order orbital 
wave fluctuations. Formally, this is done by generalize 
H120 to arbitrary psuedospin T > ^ and consider the 
limit of large T. 

First we determine the classical ground state of -ffi2o 
on a triangle cluster shown in Fig. [21 We parametrize the 



pseudospins at vertex A,B,C with angles (j)A,B,c with 
respect to the Tz axis {AB bond). The classical energy, 
E/T^ = /(O; A, B) + f{2TT/3; A, C) -h/(47r/3; B, C) where 
7(7; A, B) = cos(7 — (pA) cos(7 — 0b), is minimized when 
(t>A - (t>B ^ 4)c - 4>A ^ 27r/3, (t>A = 57r/6 or ll7r/6. 
The two energy minima manifest the C2 symmetry of 
Hi2Q, which is invariant under a 7r-rotation of T about 



the y axis, i.e. T^^; 



-Tr.z with 9 = tt/2 in The 



corresponding orbital configuration is shown in Fig. [21 
where the orbital at A forms angle 0^/2 = 75° with 
the AB bond. Permutation of {A, B, C) yields configu- 
rations of the same energy. This result immediately tells 
us the classical orbital ordering pattern (see Fig. [2]) on 
the Kagome lattice, which consists of corner sharing tri- 
angles. 

Orbital order by disorder. For the bipartite hon- 
eycomb lattice, we introduce transformation T^.z 
Tx.z = i7a;,z, with + (— ) for sites within the 
A (B) sublattice. Then the orbital exchange be- 
come ferro-orbital in terms of the new T operators. 



H12 



ReA.j 



r,(R)r,(R- 



It can be rewritten 



(apart from a constant factor) as 



H 



120 



J2 [T,(R)-T,(R-|-e,)]2,j = 1,2,3. (7) 



ReA,j 



Therefore, the classical energy is minimized for any ho- 
mogeneous configuration (T^,T'^)(R) = (Tcos0,Tsin0), 
independent of the polar angle This implies Neel (an- 
tiferro) orbital ordering. Note that this continuous S0(2) 
degenerate manifold of classical ground states is not an 
inherent property of i?i2o which is only invariant under 
finite point group rotations. Orbital wave excitations are 
important quantum fluctuations beyond mean fleld the- 
ory. To the leading order in 1/T, their correction to the 
ground state energy can be computed following the spirit 
of Holstein-Primakov spin wave theory [33| . The algebra 
is cumbersome but straightforward. The central result is 
the quantum correction to the ground state energy per 
unit cell (containing one A site and one B site). 



TJz 



(8) 



Here, index A = ±1 labels the two branches of 
the orbital wave excitations with dispersion ^^(k) — 
{V^/4)^/3 + 2X\|3k{4))\. The form factor /3k('/') = 
sm'^{(j))e^-^' + sin2(,/, - TT/3)e'^-'^^ + sin^icj) + TT/3)e'^-'^^ . 
N is the number of sites within the A sublattice, and the 
k sum is within the first Brillouin zone of the triangular 
sublattice. Ec{4>) is plotted in the right panel of Fig. [3l 
for (/) = to TT, and Ec{(j) + n) — E^cf)). We see that or- 
bital fluctuation hfts the SO (2) degeneracy and chooses 
ground state ipn = m:/3 (n is any integer), where func- 
tion Ec{4') reaches minimum. This mechanism is well 
known in frustrated spin systems as "order by disorder" 
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FIG. 3: (Color online) Left: Neel orbital ordering on the hon- 
eycomb lattice. The orbital at site A forms an angle of 30° 
with the horizontal AB bond. Global rotation of all orbitals 
by n7r/6 yields degenerate configurations. Right: The quan- 
tum fluctuation correction to the ground state energy per unit 
cell, Ec/TJz, which reaches minima at (j)n = nir/'i. This lifts 
the continuous degeneracy of the classical ground states. 



Here q = mr /ht + G \s the quasimomentum folded into 
the first Brillouin zone of the underlying lattice, G is 
the reciprocal wave vector, and /i, v are the orbital in- 
dices. The closed-shell s-fermions contribute the usual 
anti-bunching dips at G, reflecting the Fermi statistics 
fs^ , 34 1 . The p-fermions on the other hand contribute 



additional terms such as structure factor {T^T^,) and 
{T^T~,), which lead to new dips at G^, the reciprocal 
lattice wave vector of the enlarged unit cell in the ordered 
state. For example, G^ — (tt, tt), (0, tt), (27r/-\/3, 0), for 
the square, kagome, and honeycomb lattice, respectively. 

Note Added. After the submission of our manuscript, 
there appeared Ref. [s^ which independently proposed 
and studied a similar quantum 120° model. 

We thank C. Ho and V. Stojanovic for helpful discus- 
sions. This work is supported in part by ARO W911NF- 
07-1-0293. 



M [131 . The Left panel of Fig. [3] shows a representative 
orbital ordering pattern corresponding to n = 1, i.e., the 
p-orbital at site A is at angle /2 = 30° with the hori- 
zontal AB bond. Successive global rotation of all orbitals 
by 7r/6 yields degenerate Neel configurations. 

Finally we turn to the triangular lattice. We limit our- 
selves to 3-sublattice (R3) ordering [s^l as one of the 
potential orders for pseudospin T ~ 1/2 31]. The 
R3 order is parametrized by three polar angles, (j)A,B.Ci 
describing the in-plane pseudospins at each sublattice 
A,B,C. The classical energy per unit cell E/T"^ = 
3[cos(ei) -I- cos(e2) -I- cos(ei -I- e2)]/2 only depends on two 
independent parameters ei = (j)A ~ 0b and £2 = 4>b — 0c > 
and it is minimized for ei ~ €2 ~ ±27r/3. Therefore, the 
classical R3 configuration is infinitely degenerate with its 
degeneracy parametrized by (pA- Its orbital wave analysis 
will be presented elsewhere. 

To summarize, our mean-field and low energy fluctu- 
ation analysis suggests new correlated Mott phases with 
long range p-orbital ordering on the kagome, honeycomb, 
and triangular lattice. At finite temperature, we expect 
thermal fluctuations stabilize orbital ordering, as 
ously demonstrated for the cubic 120° model 
Our analysis is very suggestive but cannot be taken as 
complete for pseudospin T = 1/2. There, quantum fluc- 
tuation may be strong enough to melt the long range 
orbital order, raising the possibility of realizing a dis- 
ordered "orbital liquid" ground state. Future work is 
needed to address this open question. 

Detecting the orbital order. Here, we predict exper- 
imental signatures of the orbital-ordered phases found 
above in the density-density (noise) correlation function, 
G(r,r';t) = (n(r, t)n(r', t)) - (n(r, t))(n(r', t)), which 
can measured in the time of flight image taken after 
time t. We generalize the analysis of Ref. [l^, [H, Isl] 
to the case of multi-orbital spinless fermions and fo- 
cus on various contributions from (cj^^Cq^i/Cq/ ^/Cq'.i,'). 
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